A Framework for Multilingual Searching and Meta-information Extraction

Due in large part to the popularity and global nature of the Web, multi-lingual issues in computers is finally beginning to attract serious attention, from users and developers alike. At the Software Labs in NTT, we are involved in a project that confronts multi-lingual issues in a big way. Namely, we are building software designed to self-configure a global distributed search infrastructure. This paper describes how we have architected our system in order to handle multi-lingual issues ranging from character encoding translation to language detection to term weighting. It can serve both as a case study in the design of multi-lingual distributed systems, and as a general tutorial on multi-lingual issues. It also presents some surprising conclusions, such as the fact that the popular Unicode international character encoding is not the best one to use in our environment.